AGU 2021 Open Science in Action Tutorials
NASA Earthdata Access in the Cloud Using Open-source Libraries
Acknowledgements:
Each of the following co-authors contributed to the following materials and code examples, as well as collaboration infrastructure for the NASA Earthdata Openscapes Project:
- Julia S. Stewart Lowndes; Openscapes, NCEAS
- Erin Robinson; Openscapes, Metadata Game Changers
- Catalina M Oaida; NASA PO.DAAC, NASA Jet Propulsion Laboratory
- Luis Alberto Lopez; NASA National Snow and Ice Data Center DAAC
- Aaron Friesz; NASA Land Processes DAAC
- Andrew P Barrett; NASA National Snow and Ice Data Center DAAC
- Makhan Virdi; NASA ASDC DAAC
- Jack McNelis; NASA PO.DAAC, NASA Jet Propulsion Laboratory
Additional credit to the entire NASA Earthdata Openscapes Project community, Patrick Quinn at Element84, and to 2i2c for our Cloud computing infrastructure.
The NASA Earthdata archive continues to grow
The NASA Earthdata Cloud Evolution
NASA Distributed Active Archive Centers (DAACs) are continuing to migrate data to the Earthdata Cloud
- Supporting increased data volume as new, high-resolution remote sensing missions launch in the coming years
- Data hosted via Amazon Web Services, or AWS
- DAACs continuing to support tools, services, and tutorial resources for our user communities
Part 1: Explore Data hosted in the Earthdata Cloud
Earthdata Search Demonstration
From Earthdata Search https://search.earthdata.nasa.gov, use your Earthdata login credentials to log in. You can create an Earthdata Login account at https://urs.earthdata.nasa.gov.
In this example we are interested in the ECCO dataset, hosted by the PO.DAAC. This dataset is available from the NASA Earthdata Cloud archive hosted in AWS cloud.
Click on the “Available from AWS Cloud” filter option on the left. Here, 39 matching collections were found with the ECCO monthly SSH search, and for the time period for year 2015. The latter can be done using the calendar icon on the left under the search box. Scroll down the list of returned matches until we see the dataset of interest, in this case ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4).
View and Select Data Access Options
Clicking on the ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4) dataset, we now see a list of files (granules) that are part of the dataset (collection). We can click on the green + symbol to add a few files to our project. Here we added the first 3 listed for 2015. Then click on the green button towards the bottom that says “Download”. This will take us to another page with options to customize our download or access link(s).
Access Options
Select the “Direct Download” option to view Access options via Direct Download and from the AWS Cloud. Additional options to customize the data are also available for this dataset.
Earthdata Cloud access information
The final ordering page provides instructions to download and links for data access in the cloud. The AWS S3 Access tab provides the S3:// links, which is what we would use to access the data directly in-region (us-west-2) within the AWS cloud.
E.g.: s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc where s3 indicates data is stored in AWS S3 storage, podaac-ops-cumulus-protected is the bucket, and ECCO_L4_SSH_05DEG_MONTHLY_V4R4 is the object prefix (the latter two are also listed in the dataset collection information under Cloud Access (step 3 above)).
Integrate file links into programmatic workflow, locally or in the AWS cloud.
In the next two examples we will work programmatically in the cloud to access datasets of interest, to get us set up for further scientific analysis of choice. There are several ways to do this. One way to connect the search part of the workflow we just did in Earthdata Search to our next steps working in the cloud is to simply copy/paste the s3:// links provides in Step 4 above into a JupyterHub notebook or script in our cloud workspace, and continue the data analysis from there.
One could also copy/paste the s3:// links and save them in a text file, then open and read the text file in the notebook or script in the JupyterHub in the cloud.
Part 2: Working with Cloud-Optimized GeoTIFFs
using NASA’s Common Metadata Repository Spatio-Temporal Assett Catalog (CMR-STAC)
In this example we will access the NASA’s Harmonized Landsat Sentinel-2 (HLS) version 2 assets, which are archived in cloud optimized geoTIFF (COG) format archived by the Land Processes (LP) DAAC. The COGs can be used like any other GeoTIFF file, but have some added features that make them more efficient within the cloud data access paradigm. These features include: overviews and internal tiling.
But first, what is STAC?
SpatioTemporal Asset Catalog (STAC) is a specification that provides a common language for interpreting geospatial information in order to standardize indexing and discovering data.
The STAC specification is made up of a collection of related, yet independent specifications that when used together provide search and discovery capabilities for remote assets.
| Four STAC Specifications | |
|---|---|
| STAC Catalog (aka DAAC Archive) | STAC Collection (aka Data Product) |
| STAC Item (aka Granule) | STAC API |
CMR-STAC API
The CMR-STAC API is NASA’s implementation of the STAC API specification for all NASA data holdings within EOSDIS. The current implementation does not allow for querries accross the entire NASA catalog. Users must execute searches within provider catalogs (e.g., LPCLOUD) to find the STAC Items they are searching for. All the providers can be found at the CMR-STAC endpoint here: https://cmr.earthdata.nasa.gov/stac/.
In this example, we will query the LPCLOUD provider to identify STAC Items from the Harmonized Landsat Sentinel-2 (HLS) collection that fall within our region of interest (ROI) and within our specified time range.